1. Field of the Invention
The present invention relates generally to digital image display systems and more particularly to using temporal gamma and reverse super-resolution techniques to process images for optimizing appearance quality.
2. Discussion of Prior Art
Cathode Ray Tubes (CRTs), used in conventional televisions and computer monitors, are analog devices which scan an electron beam across a phosphor screen to produce an image. Digital image processing products that enhance display graphics and video on CRTs have been increasingly available because CRTs can work with many different input and output data formats. Further, CRTs can display moving images with high quality screen brightness and response. However, CRTs have considerable limitations in applications such as portable flat screen displays where size and power are important. Additionally, as direct view CRT display size increases, achieving high quality completely across the display becomes more difficult and expensive.
Many recent portable and desktop systems include digital displays using liquid crystal displays (LCDs), a term which generally describes flat-panel display technologies and in particular active matrix liquid crystal displays (AMLCDs), silicon reflective LCDs (si-RLCDs), ferroelectric displays (FLCs), field emission displays (FEDs), electroluminescent displays (ELDs), plasma displays (PDs), and digital mirror displays (DMDs).
Compared to traditional CRT displays LCDs have the advantages of being smaller and lighter, consuming less power, and having discrete display elements, which can provide consistent images across the entire display. However, manufacturing LCDs requires special processing steps to achieve acceptable visual quality. Further, large screen direct view LCDs are expensive, and LCDs usually require a display memory.
Both CRT and LCD technologies can provide economical projection system large screen displays. CRT-based projection systems usually require three CRTs and three projection tubes, one for each of the Red (R), Green (G), and Blue (B) color components. Each tube must produce the full resolution display output at an acceptable brightness level, which makes the tubes expensive. Achieving proper tolerances for mechanical components in projection systems, including alignment hardware and lenses, is also expensive. Consequently, manufacturing CRT-based projection systems is costly. Since CRTs are analog, applying digital image processing techniques to CRT-based systems usually requires a frame buffer memory to effectively represent the digital image data.
Projection display systems also use transmissive or reflective LCD xe2x80x9cmicrodisplayxe2x80x9d technologies. Achieving the desired full color gamut in LCD-based parallel color projection systems, as in CRT-based projection systems, uses three separate LCD image modulators, one for each of the R, G, and B color components. A single LCD image modulator which produces R, G, and B either through spatial color filters or with sequential color fields at a sufficiently high rate can provide a low cost system.
FIG. 1 shows a prior art projection system 150 that includes a light system 100, mirrors 102, 104, 106, and 108, transmissive image modulators 110, 112, and 114, dichroic recombiners 116 and 118, and a projection lens 120. Light system 100 includes an illumination source such as a xenon lamp and a reflector system (not shown) for focusing light.
Mirrors 102, 104, 106, and 108, together with other components (not shown) constitute a separation subsystem that separates the light system 100 output white light beam into color components Red (R), Green (G), and Blue (B). The separation subsystem can also use prisms, including x-cube dichroic prism pairs or polarizing beam splitters.
Each image modulator 110, 112, and 114 receives a corresponding separated R, G, or B color component and functions as an active, full resolution, monochrome light valve that, according to the desired output images, modulates light intensities for the respective R, G, or B color component. Each image modulator 110, 112, and 114 can include a buffer memory and associated digital processing unit (not shown). A projection system may use only one image modulator which is responsible for all three color components, but the three image modulator system 150 provides better chromaticity and is more efficient.
Dichroic recombiners 116 and 118 combine modulated R, G, and B color components to provide color images to projection lens 120, which focuses and projects images onto a screen (not shown).
FIG. 1 system 150 can use transmissive light valve technology which passes light on axis 1002 through an LCD shutter matrix (not shown). Alternatively, system 150 can use reflective light valve technology (referred to as reflective displays) which reflects light off of digital display mirror display (DMD) image modulators 110, 112, and 114. Because each image modulator 110, 112, and 114 functions as an active, full resolution, monochrome light valve that modulates the corresponding color component, system 150 requires significant buffer memory and digital image processing capability.
Because of inherent differences in the physical responses of CRT and LCD materials, LCD-based projection and direct view display systems both have different flicker characteristics and exhibit different motion artifacts than CRT-based display systems. Additionally, an intense short pulse depends on the properties of CRT phosphors to excite a CRT pixel whereas a constant external light source is intensity modulated during the frame period of an LCD display. Further, LCDs switch in the finite time it takes to change the state of a pixel. Active matrix thin film transistor (TFT) displays, which have an active transistor controlling each display pixel, still require a switching time related to the LCD material composition and thickness, and the techniques of switching.
Most LCD-based image modulators (110, 112, 114, etc.) are addressed in raster scan fashion and require refreshing each pixel during each display frame interval. Accordingly, every output pixel is written to the display during every refresh cycle regardless of whether the value of the pixel has changed since the last cycle. In contrast, active matrix display technologies and some plasma display panel technologies allow random access to the display pixels. Other, simpler panels use a simpler row by row addressing scheme similar to the raster scan of a CRT. Additionally, some displays have internal storage to enable output frames to self-refresh based on residual data from the previous output frame.
Field Emission Displays (FEDs) may include thousands of microtips grouped in several tens of mesh cells for each pixel. The field emission cathodes in FEDs can directly address sets of row or column electrodes in FEDs, and FEDs have fast response times. FEDs can use external mesh addressing for better resolution images, but this requires increased input/output (I/O) bandwidth outside of the FED.
Opto-mechanical systems can provide uniform brightness and high chromaticity for high quality displays. Additionally, high quality projection lens systems can provide bright and uniform images. However, component and assembly tolerances in opto-mechanical systems can result in system imperfections including imprecise image modulator alignment and geometric lens distortion.
Commercially available digital image processing systems, usually a part of an electronic control subsystem, can process analog or digital input data and format the data into higher resolution output modes. These processing systems typically perform operations such as de-interlacing, format conversion and line doubling or quadrupling for interlaced analog input data. Some systems include a decompression engine for decompressing compressed digital data, and input data scaling to match the resolution and aspect ratio to the display device. However, these systems do not perform advanced image processing specific to a digital imaging LCD or to the display system. Additionally, these digital image processing systems do not often accommodate digital or compressed digital image data which can include bitstream information for enhanced outputs.
Image sensing algorithms, for example in remote sensing and computer vision applications, use special sampling and image warping techniques to correct input sensor distortions and to reconstruct images.
Data compression tools such as those standardized by the Moving Pictures Experts Group (MPEG) can compact video data prior to transmission and reconstruct it upon reception. MPEG-2 can be applied to both standard definition (SDTV) and high definition television (HDTV).
Projecting an image from a projector on a tabletop to a flat screen which is closer to the projector at the bottom than the top results in an image which is narrower at the bottom than at the top in what is known as the xe2x80x9cKeystonexe2x80x9d effect.
Radial distortion occurs when an image pixel is displaced from its ideal position along a radial axis of the image. Because an image has the largest field angles in the display corners, the corners exhibit worse radial distortion than other display areas. Radial distortion includes barrel distortion, where image magnification decreases towards the corners, and pin cushion distortion, where the magnification increases towards the corners. Lens related distortions including radial distortion can cause image deformation. Distortion can also result from non-flat screens or earth""s magnetic field.
Image modulators (110, 112, 114, etc.) have a fixed number of pixels spaced uniformly in a pattern. Projecting an image from an image modulator to a display screen deforms the uniformity of pixel spacing, that is, pixels are not correlated one to one from the image modulator to the display screen. Therefore, some screen display regions have more image modulator pixels than screen pixels while other screen display regions have fewer image modulator pixels than screen pixels.
Motion artifacts appear where image objects move near the edges of curved screens. Even when a flat screen projection is motion-adaptive filtered, the difference in the distances of objects from the projector causes an apparent motion of moving objects on a curved screen. Additionally, extremely large curved screens can achieve necessary resolution and brightness only with film projectors.
Multiple camera systems are commonly used to improve display quality on curved screen displays. For example, two cameras record respective halves of a scene to improve output. A layered coding technique may include a standard MPEG-2 stream as a base layer and enhancement information as a supplemental layer. Even if the two views are from slightly different angles, the compression ratio for the two camera views combined is less than the total compression ratio would be if each view were captured and compressed independently. Additionally, the second camera can provide a view that may be occluded from the first camera. Systems using additional camera angles for different views can provide additional coded and compressed data for later use. Multiple camera systems can also compensate for the limited focal depth of a single camera and can substitute for the use of a depth finding sensor which senses and records depth information for scenes. Image processing can improve the outputs of multiple camera systems.
Stereoscopic photography also uses multi-camera systems in which a first camera records a left eye view and a second camera records a right eye view. Because camera lenses focus at a certain distance, one camera uses one focal plane for all objects in a scene. A multi-camera system can use multiple cameras each to capture a different focal plane of a single scene. This effectively increases the focal depth. Digital image processing can further improve focusing for these multi-camera systems.
Types of three dimensional binocular display systems include anaglyph displays, frame sequence displays, autostereoscopic displays, single and multi-turn helix displays. These normally have multiple camera data channels. Anaglyph systems usually require a user to wear red and green glasses so that each eye perceives a different view. Frame sequencing systems use shutter glasses to separate left and right views. Autostrereoscopic displays use lenticular lenses and holographic optical elements. Single or multi-turn helix displays use multiple semi-transparent display screens which can be seen by multiple observers without special glasses. Multiple camera data channels systems can benefit from image processing.
Each R, G, and B color component has different intensity values which are digitally represented by a number of bits. For example, if 8 bits represent each R, G, and B color component then each component has 256 (=28) intensity values from 0 to 255. Changing intensity value of a color component in an ideal digital device from a number X, for example, to a number Y, takes just as long whatever the Y value. Consequently, changing a color component value from 2 to 3 takes as long as changing the value from 2 to 200. However, because of the nature of LCD image modulator pixels, the transitions for modulating light intensities are not purely digital, and various analog distortions remain.
What is needed is an image processing system to enhance display quality and provide the best possible visual images.
The present invention is directed to an image processing system for enhancing visual quality and improving performance in digital display systems. The processing system, useful for DTV displays and electronic theaters, can receive different types of data inputs including analog, digital, and compressed (or coded) display images. When lower resolution input images are used, the images are converted into higher quality display images using multiframe restoration. The processing system uses known information to provide unique enhancements to the image modulator and to the display system, and thus takes into account system characterization, manufacturing defects, calibration data, environment effects, and user controlled setup information. The processing system also employs Temporal Gamma Processing (TGP) and Reverse Super-Resolution (RSR) techniques to process images.
TGP independently processes each R, G, or B color component and compensates for modulating transition characteristics to assure that the time related representation of an image is as accurate as possible. TGP uses a previous frame value and a known transfer function of the display modulating system to adjust TGP outputs to desired values. TGP can overdrive an LCD image modulator to compensate for the LCD material characteristics so that the desired output value can be quickly achieved.
RSR performs a superset of the frame rate conversion process for converting disparate input and output frame rates. RSR improves display quality when intended display images have a higher apparent resolution than can be supported by the number of pixels of the image modulator. RSR, block by block, spatially filters one frame in a video sequence having a higher resolution into numerous frames having a higher frame rate and a lower resolution than the original frame. RSR, while sequencing the higher frame rate and lower resolution frames, shifts the pixel matrix of each sequenced frame and uses motion tracking to filter the images and to prevent motion artifacts. RSR can also be used with a moveable image modulator that shifts the pixel matrix by a sub-pixel distance during each sequenced frame refresh, that is, during each high resolution image the frame is sequenced into lower resolution frames and the frames are displayed during the cycle of frame refreshes.